Papers with prompt-based fine-tuning
DynaMaR: Dynamic Prompt with Mask Token Representation (2022.emnlp-industry)
Copied to clipboard
Xiaodi Sun, Sunny Rajagopalan, Priyanka Nigam, Weiyi Lu, Yi Xu, Iman Keivanloo, Belinda Zeng, Trishul Chilimbi
| Challenge: | Recent research shows that large language models pretrained using unsupervised approaches can achieve significant performance improvement on many downstream tasks. |
| Approach: | They propose an unsupervised approach to fine-tuning large language models using unsupervised approaches to many downstream tasks. |
| Outcome: | The proposed approach improves on four e-commerce applications and can achieve an average improvement of 10% in few-shot settings and 3.7% in data-rich settings over the standard approach. |
Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks. |
| Approach: | They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks. |
| Outcome: | Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks. |
LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models have been limited when fine-tuned on small datasets. |
| Approach: | They propose to add contrastive learning to prompt-based fine-tuning to improve model performance. |
| Outcome: | The proposed approach outperforms other methods on multiple text classification benchmarks. |
FPC: Fine-tuning with Prompt Curriculum for Relation Extraction (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for relation extraction ignore semantics of relation labels . prompt-based fine-tuning has been proposed for RE . |
| Approach: | They propose a method for relation extraction using prompt-based fine-tuning . they use auxiliary prompt-tuned learning task to make the model capture semantics of relation labels . |
| Outcome: | The proposed method outperforms existing methods on four widely used RE benchmarks under fully supervised and low-resource settings. |
Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent few-shot learning methods focus on improving downstream task performance, but there is limited understanding of the adversarial robustness of such methods. |
| Approach: | They evaluate prompt-based FSL methods against fully fine-tuned models to better understand the impact of various factors towards robustness. |
| Outcome: | The proposed methods show that they are less robust in the face of adversarial perturbations than fully fine-tuned models. |
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners (2022.findings-naacl)
Copied to clipboard
| Challenge: | LiST is an efficient method for fine-tuning large pre-trained language models in few-shot learning settings. |
| Approach: | They propose a method for efficient fine-tuning of large pre-trained language models in few-shot settings using self-training and meta-learning. |
| Outcome: | The proposed method outperforms GPT-3 in-context learning by 33% on few-shot tasks. |
Making Pre-trained Language Models Better Few-shot Learners (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that the GPT-3 model can perform few-shots on language understanding tasks with a natural-language prompt and a few task demonstrations. |
| Approach: | They propose a technique for fine-tuning language models using a few examples . they propose LM-BFF, which uses prompt-based fine-uning and a pipeline for automating prompt generation . |
| Outcome: | The proposed approach outperforms standard fine-tuning procedures on a range of NLP tasks. |
SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have used prompt-based fine-tuning methods for text classification tasks . however, the difficulty and costs of manually selecting domain label terms for the verbalizer remain unexplored . |
| Approach: | They propose a framework to automatically retrieve scientific topic-related terms for low-resource text classification tasks. |
| Outcome: | The proposed method outperforms state-of-the-art methods on scientific text classification tasks under few and zero-shot settings. |
Comparing Prompt-Based and Standard Fine-Tuning for Urdu Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have demonstrated the efficacy of pre-trained language models for various downstream tasks. |
| Approach: | They compare prompt-based fine-tuning with standard fine-uning for text classification in Urdu and Roman Urdu languages. |
| Outcome: | The proposed approach improves up to 13% in accuracy in low-resource languages with limited labeled examples over standard fine-tuning approaches. |
Adaptive Cross-lingual Text Classification through In-Context One-Shot Demonstrations (2024.naacl-long)
Copied to clipboard
| Challenge: | Zero-Shot Cross-lingual transfer (ZS-XLT) uses a model trained in a source language to make predictions in another language, often with a performance loss. |
| Approach: | They propose a new approach that uses In-Context Tuning to train a model to learn from context examples and adapt it to a target language by prepending a One-Shot context demonstration. |
| Outcome: | The proposed approach outperforms prompt-based models in Zero-Shot and Few-shot scenarios with target-language examples. |
SMASH: Improving SMAll Language Models’ Few-SHot Ability with Prompt-Based Distillation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models with prompts have shown remarkable performance on few-shot learning. |
| Approach: | They propose an approach to improve SMAll language models’ few-SHot ability by training on intermediate tasks before prompt-based fine-tuning on downstream tasks. |
| Outcome: | The proposed model improves on sentence-pair and sentiment classification tasks by training on intermediate tasks before fine-tuning on downstream tasks. |
Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based learning is susceptible to intrinsic bias present in pre-trained language models (LMs), leading to sub-optimal performance in prompt-based zero/few-shot settings. |
| Approach: | They propose a null-input prompting method to calibrate intrinsic bias encoded in pre-trained language models (LMs) they leverage a diverse set of auto-selected null meaning inputs generated from GPT-4 to probe intrinsic bias. |
| Outcome: | The proposed method significantly improves zero/few-shot learning performance of LMs for both in-context learning and prompt-based fine-tuning (on average 9% and 2%, respectively). |
Adversarial Knowledge Stimulated Contrastive Prompting for Few-shot Language Learners (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained language models on few-shot Natural Language Understanding (NLU) tasks by employing task-specific prompts. |
| Approach: | They propose a Cloze-driven prompt framework for prompt tuning that implicitly stimulates knowledge from pre-trained language models. |
| Outcome: | The proposed framework outperforms state-of-the-art for prompt-based fine-tuning on few-shot NLU tasks. |
Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent prompt-based fine-tuning techniques have garnered considerable interest as a core technique for few-shot text classification tasks. |
| Approach: | They propose a prompt-based fine-tuning approach that reformulates the fine-uning objective to align with the Masked Language Modeling objective. |
| Outcome: | The proposed method has shown superior performance on five multi-class classification datasets. |